|
Poster:
|
Brad Leblanc |
Date:
|
Jan 14, 2006 1:04am |
|
Forum:
|
etree
|
Subject:
|
Information about XML (metadata) files and Deriver |
Hi folks, in the interest of educating our patrons, curators thought it would be a good idea to start documenting and communicating some of the more technical information about the contribution system. It is currently a slightly complex system and takes some getting used to. At some point, we’ll probably find a permanent and easier to find home for this information, but for now we think posting it to the forum is better than not sharing it.
In this post we will cover the "XML" files, the "rules.conf" file and how they relate to lossies getting created and the deriver.
Stage 1:
When you import a new recording, you begin by filling out an "import form" to describe the concert. In this form you will receive the option of disallowing lossy derivatives. We call them derivatives because the original lossless FLAC or SHN files are used to "derive" the Ogg and MP3 files that you see on many of the recordings. The engine that performs this work is aptly titled the "deriver". The option of allowing/disallowing lossies presents itself at the bottom of the import screen as seen in attached
image1.jpg.
Occasionally you may notice that this option has been disabled due to the artist’s request that lossies *never* get created. (see
image2.jpg)
When you submit this first form, it will create 2 files from the info you have provided:
1. [itemID]_meta.xml (
mandatory for the item to continue processing)
2. [itemID]_rules.conf (
optional for the item to continue processing – only created if the person uploading the recording or the artist opt out of having lossy files)
Where [itemID] = the name of the folder you used for the recording being imported.
The [itemID]_meta.xml file contains all the information you submit in the form - including source, lineage, taped by, transferred by, setlists etc… This information is used to "paint" the details page that other patrons will browse in the future.
The [itemID}_rules.conf file will only be created if you opt-out of the lossy formats. This is done by checking the box we discussed above. The content of this file is the text string "CAT.lossy" which tells the deriver not to create lossy files, and only create the full show zip.
Stage 2:
Following the submittal of the above form is the form called "File Options" in which you describe all of the files you have contributed. -- Is the file a FLAC file or SHN? Is it CD Quality or 24bit audio? Is the file text? If the file is a music file, give the song a title and the artist a name. If the file is text information then you should identify the author in the "creator" field – usually the taper’s name. If the file is music, the "creator" field should auto fill with the name of the artist in the recording. Submit this information when you have confirmed all of it is accurate to the best of your knowledge. See
image3.jpg for a copy of this form.
This information creates another XML file:
1. [itemID]_files.xml (
mandatory for the item to continue processing)
If you have allowed lossy files to be created, the information you entered into this form will be used to write ID tags on the lossy files. This means that media players such as iTunes, WinAmp, Windows Media Player etc… will show artist name, album info, and song names automatically when the files are played. However, this is a 1 shot deal – if you enter the information here incorrectly, updating it after the lossy files are created will not update the tags on these files. The incorrect tags will remain until the derivatives are regenerated, so it is very important to get this right the first time.
Stage 3:
Once the system notices that both the mandatory files exist - [itemID]_files.xml and [itemID]_meta.xml - (it sweeps for them about once every 10 minutes), it will move the item on to the next stage of the process – the deriver. The files will be copied to a "worker" machine which will do the following:
- verify that both mandatory metadata files exist and the information is valid (no
illegal characters)
- verify the file integrity of the audio (run included MD5’s, FLAC checks)
- check for presence of [itemID]_rules.conf
- create lossy files if no [itemID]_rules.conf was found.
- tag lossy files
- copy all finished files back to the upload server
- change the status to "Waiting for curator approval".
This post was modified by Brad Leblanc on 2006-01-14 09:04:51
Attachment: image1.jpgAttachment: image2.jpgAttachment: image3.jpg
Live Music Archive
Librivox Free Audio
Metropolitan Museum
Cleveland Museum of Art
Internet Arcade
Console Living Room
Books to Borrow
Open Library
TV News
Understanding 9/11
